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I. Real Party in Interest 

The real party in interest with respect to this application is Hewlett-Packard. 

II. Related Appeals and Interferences 

There are no other prior and pending appeals, interferences or judicial 
proceedings known to the Appellants, the Appellants' legal representative, or the 
assignee which may be related to, directly affect or be directly affected by or have a 
bearing on the Board's decision in this pending appeal. 

III. Status of Claims 

The claims currently pending in this application are Claims 1-20, all of which 
stand finally rejected. Claims 1-20 are being appealed. 

IV. Status of Amendments 

No amendments were filed after final rejection. 

V. Summary of Claimed Subject Matter 

The Hirata patent, considered alone, or in combination with the various 
secondary references relied upon by the Examiner, fails to teach or suggest all 
features set forth in Appellants' independent claims 1 and 17. For example, the 
Hirata patent, considered alone, or in combination with the various secondary 
references relied upon by the Examiner, does not redisplay a portion of the images 
as claimed, and does not suggest redisplay of a portion of the images first displayed. 
The applied references do not disclose or suggest a first visual display of a plurality 
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of images, and redisplaying a portion of the first displayed images, the portion being 
in reference to the first display. The independent claims encompassing at least 
these and other salient features are summarized below. 

Claim 1 encompasses a method of visualizing and retrieving a data file from a 
set of data files, e.g., as illustrated in Fig. 1, comprising, for example, displaying a 
plurality of images 110 representing corresponding data files on a display device 
using a first distance metric between each data file (e.g., page 5, paragraph [001 1], 
lines 20-22), redisplaying a portion of the images 120 on the display device using a 
refined distance metric (e.g., page 6, paragraph [0011], lines 1-3), and performing at 
least one of retrieving 140, marking, and selecting at least one desired data file (e.g., 
page 6, paragraph [001 1], lines 3-13). 

Claim 17 encompasses a method of interactively retrieving a data file from a 
set of data files in real time, e.g., as illustrated in Fig. 3, comprising, for example, 
displaying a plurality of images 310, each image corresponding to a data file, on a 
display device using a first distance metric between each data file, interactively 
selecting 320, by a user, a portion of the images, redisplaying 330 the portion of the 
images in real time on the display device using a refined distance metric, and 
retrieving a desired data file 350 (e.g., paragraphs [0018] and [0019]). 

VI. The Issues 

The final Office Action presents the following issues for a review on this 
appeal: 

A. Whether claims 1-7, 11,12 and 16-19 are anticipated by U.S. 
Patent No. 6,240,423 (Hirata); 



Appeal Brief 
Application No. 09/904,627 
Attorney's Docket No. 10006286-1 

Page 4 

B. Whether claims 8, 13 and 14 are unpatentable over the Hirata 
patent, and further in view of U.S. Patent 6,584,221 (Moghaddam et al.); 

C. Whether claims 9, 10 and 20 are unpatentable over the Hirata 
patent, and further in view of U.S. Patent 6,121 ,969 (Jain et al.); and 

D. Whether claim 15 is unpatentable over the Hirata patent, and 
further in view of U.S. Patent 5,528,259 (Bates et al.). 

VII. Argument 

A. The Hirata Patent Does Not Disclose Each And Every Element 
Recited In The Claims. 

The rejection based on the Hirata patent cannot stand because the Hirata 
patent fails to either expressly or inherently disclose a number of features recited in 
independent claims 1 and 17. Additionally, the dependent claims recite 
combinations of features defining combinations of separately patentable subject 
matter not disclosed in the Hirata patent. To anticipate the claimed invention, the 
cited document must disclose each and every feature set forth in the claimed 
combination of features (see, MPEP § 2131). 

Independent claims 1 and 17 recite, among other features, redisplaying a 
portion of the images on the display device using a refined distance metric. In 
contrast, the Hirata patent is directed to a query based image matching system, 
wherein images are displayed only a single time per query. Using this method, the 
Hirata patent displays the results of a database query based upon results of a 
combined region based image matching and a boundary based image matching. 
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With respect to claim 1, the Examiner asserts on pages 2 and 3 (paragraph 3) 

of the final Office Action that Hirata patent discloses: 

Displaying a plurality of images representing corresponding data files 
on a display device using a first distance metric (Calculating a first 
similarity between a query image and images in the database using a 
region-based matching to produce a first set of similar images. Images 
being retrieved in order based upon the distance between the query 
image and the result candidate images, column 6) between each data 
file (e.g., Based on the mutual similarities between the images, retrieval 
results of the candidate images are sorted. The images which are 
similar to each other are assumed to be under one group and are re- 
ordered based on the similarity and users can specify the forms of the 
output for display; figures 1-11; column 6-9); 

Redisplaying a portion of the images on the display device using a 
refined distance metric (e.g., Hirata teaches using a distance metric 
such as a second similarity to retrieve images based on the set of 
similar images retrieved using a first similarity. The second similarity is 
now a "refined" distance metric because it refines the search for 
images. The second similarity of Hirata thus meets the claim limitation 
of a "refined distance metric." Here, a refined set of similar images is 
produced from the first set of similar images produced using a first 
distance metric, figures. 9, 11 and 12; column 6-9); and 

Performing at least one of retrieving, marking and selecting at least one 
desired data file (Hirata teaches retrieving and selecting at least one 
desired data file; see, figures 1-11; column 6-9). (Underlining and 
italics in original.) 



With respect to claim 17, on page 6 of the Office Action, in numbered 

paragraph 4, the Examiner asserts that the Hirata patent discloses clustering image 

data files using a distance metric such as a first similarity, with reference to Figures 

1-11 and at column 6-9. The Examiner also asserts that the Hirata patent discloses: 

Interactively selecting, by a user, a portion of the images (e.g., a user 
[specifies] the regions of the images; e.g., the region division of figure 
3; figures 1-11; column 6-9); 

Redisplaying the portion of the images in real time on the display 
device using a redefined distance metric (e.g., Hirata teaches using a 
distance metric such as a second similarity to retrieve images based on 
the set of similar images retrieved using a first similarity. The second 
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similarity is now a "refined" distance metric because it refines the 
search for images. The second similarity of Hirata thus meets the 
claim limitation of "a refined distance metric." Here, a refined set of 
similar images is produced from the first set of similar images produced 
using a first distance metric. Re-order based on the similarity among 
the candidate images of figure 9 or 2 nd stage image matching based on 
boundary of figure 11 and grouping and re-ordering is based on the 
similarity among candidates using the refined distance metric of figure 
1 1 ; figures 1-11; column 4); and 

Retrieving a desired data file (retrieved data file is displayed on a 
display device; figures 1-11; column 4). 



The foregoing rejections are respectfully traversed, as the Hirata patent, 
considered alone, or in combination with the various secondary references relied 
upon by the Examiner, fails to teach or suggest all features set forth in Appellants' 
independent claims 1 and 17. 

The present application discloses a multi-media database and classification 
system, which can, for example,, provide for automatic classification and retrieval of 
multimedia files based on features of the multimedia files. Figure 1 shows an 
exemplary method for visualizing and retrieving data files in accordance with an 
exemplary embodiment of the present invention. In step 1 10, a plurality of images 
representing data files on a display device are displayed using a first (i.e., coarse) 
distance metric. The distance metric represents a distance between each data file. 
In step 120, a portion of the images can be redisplayed on the display device using 
a refined distance metric. As described on specification page 6 beginning with line 
4, the distance metric can be refined at each of plural redisplays until a desired data 
file is found or a maximum refined distance metric is reached. In step 140, a desired 
data file can be retrieved, and/or can be marked or selected. 
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Figures 2A-2E show a graphical representation of displays generated in 
accordance with exemplary embodiments, where each portion of the images 
redisplayed can be graphically selected by a user. Figure 2A is a screen capture of 
an exemplary display showing a first level of images representing data files on a two- 
dimensional display. A first, coarse, distance metric can be calculated, and allows 
the user to receive useful information about the organization of images on the 
display. Based on perceived properties of a desired image, a user can select an 
area 202 of the screen where a desired image most likely resides. Figure 2B shows 
a portion of images redisplayed as selected by area 202. Here the distance metric 
has been recalculated using more of the image information than was used in the first 
distance calculation. 

The redisplay process can be repeated, as represented by Figures 2C and 
2D. Distance metrics between images are again recalculated using more image 
feature data than previously used with respect to Figure 2B. Following the redisplay 
based on recalculated distance metrics, a desired data file (e.g., image 240) is 
identifiable and can be selected and retrieved as shown in Figure 2E. Figure 3 
shows another exemplary method which can be implemented in accordance with the 
present invention, and Figure 4 shows a flowchart for performing a coarse to fine 
distance calculation. 

The foregoing features are broadly encompassed by independent claims 1 
and 17. In contrast to using query based image retrieval, exemplary embodiments of 
the present invention display images using a first distance metric between each 
data file, and provide for redisplaying a portion of the images using a refined 
distance metric. The claim 1 method includes a step of displaying a plurality of 



Appeal Brief 
Application No. 09/904,627 
Attorney's Docket No. 10006286-1 

Page 8 

images representing corresponding data files on a display device using a first 
distance metric between each data file. Claim 1 also recites redisplaying a portion 
of the images on the display device using a refined distance metric. Claim 17 is 
directed to a method of interactively retrieving a data file from a set of data files in 
real time, and includes features similar to those mentioned with respect to claim 1 . 

The foregoing features are neither taught nor suggested by the Hirata patent 
considered individually or in combination with the various secondary documents 
relied upon by the Examiner. 

In contrast to the presently claimed invention, the Hirata patent is directed to a 
method for querying a database of images and displaying the results in a manner as 
illustrated, for example, in Figure 10B of the Hirata patent. The results of the query 
are based upon a combination of region based image matching and boundary based 
image matching as described, for example, at column 2, lines 48-65. In the Hirata 
patent's query-based system, an initial query is submitted and a set of results is 
clustered. The clustering of the search results is performed using a distance 
between each file and a distance to the query, as discussed at column 6, lines 37- 
41 . In all cases, clustering involves, at least in part, using the distance to a query 
(see the Hirata patent's Figures 8, 9 and 10). The Hirata patent is not directed to 
redisplaying a portion of displayed images using a refined distance metric. In 
contrast to Appellants' claimed use of a refined distance metric to redisplay a portion 
of displayed images, Hirata teaches applying a new distance metric to all images 
during each image matching operation. 
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Thus, in contrast to the method disclosed and recited in Appellants' 
independent claims 1 and 17, the Hirata patent fails to teach or suggest using a 
refined distance metric to redisplay a portion of displayed images. 

In addition, the Hirata patent fails to teach or suggest the claim 17 feature of 
interactive user selection for selecting a subset of images to be displayed. Rather, 
the query based system of the Hirata patent groups all images, and only creates new 
groupings based on a new query. The Hirata patent does not teach or suggest 
applying a refined distance metric in a second stage of image matching to redisplay 
a portion of displayed images. 

In rejecting claim 17 on page 6 of the Office Action, the Examiner refers to the 
region division in Figure 3 of the Hirata patent as illustrating user specified regions. 
However, Hirata does not teach or suggest that a refined distance metric, as 
presently claimed, is used to redisplay the images of any defined region (see Hirata 
at column 4, line 10 to column 5, line 13). As such, the Hirata patent fails to teach or 
suggest the user interactive selection feature of Appellants 1 claim 17. 

Thus, the Hirata patent fails to teach or suggest the features recited in 
Appellants' independent claims 1 and 1.7. 

As such, independent claims 1 and 17 are allowable. All of the other claims 
depend from independent claims 1 and 17 and recite additional advantageous 
features which further distinguish over the documents relied upon by the Examiner. 
Hence, for these additional reasons, the Hirata patent fails to disclose each and 
every limitation set forth in independent claims 1 and 17, and hence also in claims 
depending from these independent claims. 
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For at least these reasons, the rejection of independent claims 1 and 17 is 
improper. Accordingly, the rejection should be reversed. 

B. The Hi rata Patent And The Secondary References, Fail To Teach Or 
Suggest All Features Of Claims 1 and 17 

The Hirata patent, considered alone, or in combination with the various 
secondary references relied upon by the Examiner, fails to teach or suggest all 
features set forth in Appellants' independent claims 1 and 17. Specifically, the Hirata 
patent fails to teach or suggest at least the above argued features of redisplaying a 
portion of the images on the display device using a refined distance metric recited in 
Appellants' independent claims 1 and 17. In contrast, the Hirata patent is directed to 
a query based image matching system, wherein images are displayed only a single 
time per query. 

The patents to Moghaddam et al., Jain et al. and Bates et aL, as relied upon 
by the Examiner, fail to overcome the noted deficiencies of the Hirata patent. For 
example, the Moghaddarm et al. patent is directed to a method for representing an 
image in an image retrieval database, further disclosing separation and filtering of 
given images to extract color and texture features; the Jain et al. patent is directed to 
a visual navigation in perceptual databases, the given images being ranked and 
correlated to allow perception of selected images and the relationship between the 
images; and the Bates et al. patent discloses multi-dimensional scrolling of 
overlapping data collections which are displayed in multiple layers. The patents to 
Moghaddam et al., Jain et al. and Bates et al., as relied upon by the Examiner, do 
not disclose or suggest at least the above argued features of redisplaying a portion 
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of the images on the display device using a refined distance metric recited in 
Appellants 1 independent claims 1 and 17. 

As such, independent claims 1 and 17 are allowable. Claims 8-10 and 13-15 
depend from claim 1, and claim 20 depends from claim 17, and recite additional 
advantageous features which further distinguish over the documents relied upon by 
the Examiner. 

Claims 8, 13 and 14 stand rejected under 35 U.S.C. § 103(a) as being 
unpatentable over the Hirata patent, and further in view of U.S. Patent 6,584,221 
(Moghaddam et al.); claims 9, 10 and 20 stand rejected under 35 U.S.C. § 102(e) as 
being unpatentable over the Hirata patent, and further in view of U.S. Patent 
6,121,969 (Jain et al.); and claim 15 stands rejected under 35 U.S.C. § 103(a) as 
being unpatentable over the Hirata patent, and further in view of U.S. Patent 
5,528,259 (Bates et al.). 

For at least these reasons, the Hirata patent does not disclose or suggest 
each and every element of claims 8-10, 13-15 and 20, and the patents to 
Moghaddam et al., Jain et al. and Bates et al. do not make up for the deficiencies of 
the Hirata patent. Accordingly, the rejection should be reversed. 

C. The Examiner's Assertions In The Advisory Action Are Traversed 
Because The Hirata Patent Fails To Disclose Or Suggest Each And 
Every Element Recited In The Claims. 

The Examiner responds to these points at page 2 of the Advisory Action: 

1) Applicant argues in essence that the Hirata patent is not directed to 
redisplaying a portion of displayed images using a refined distance 
metric. In response, Hirata teaches in column 7 that the step of image 
matching based on boundary is applied to the results of image 
matching based on region and the step of grouping and re-ordering 
based on similarity among result candidates is applied to the results of 
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the image matching based on boundary. Moreover, in column 14 or in 
the claims 37-39, Hirata clearly teaches displaying the second set of 
images grouped by the clusters on the display and REDISPLAYING 
RE-ORDERED SECOND SET of IMAGES ON THE DISPLAY and 
therefore at least two steps of displaying is involved in the process. 

2) Moreover, Hirata teaches in column 6 a [similarly] matching among 
the candidate images resulting from the region-based and boundary- 
based matching. Not only users can select regions for the image 
matching or querying, but also users can specify other parameters 
such as the threshold values to determine how the images are 
[categorized] into groups and this categorization is used to REDUCE 
the number of images to be displayed. 

3) Hirata discloses in column 7 that the number of results (the number 
of the input images for the step (3) as taught in Hirata) to be processed 
in the step of image matching based on boundary and the step of 
grouping and re-ordering based on similarity among result candidate 
images can be CONTROLLED BY THE USER, and such a selection 
further limits the number of the candidate images to be re-displayed in 
step (3). 

4) The displaying and re-displaying of the resulting candidate images 
are [interactive] because users can select the number of the candidate 
images to be processed and the threshold values for the clustering of 
the candidate images, thereby determining how the portion of 
candidate images (which is less than the candidate images from step 
(1) and (2) as taught in column 7 of Hirata) are arranged on the display. 
By selecting the number of the candidate images in the step of 
grouping and re-ordering based on similarity among the result 
candidates and the selection of the representative image in each 
cluster, the resulting images being redisplayed is less than the set of 
the resulting candidates from steps (1) and (2). Hirata thus teaches 
redisplaying a portion of displayed images selected in the previous step 
of the boundary image matching using a mutual similarity measure. 
Users' selection of the candidate images from the steps (1) and (2) 
produces a refined set of the candidate images for the step (3). Finally, 
it is noted that the mutual similarity as calculated in Hirata's column 9, 
lines 1 — the claim limitation of a distance measure. 

5) From the teaching of column 6-7 and 14, it is concluded that Hirata 
at least discloses two separate steps of displaying the candidate 
images wherein a lesser number of the clustered candidate images can 
be re-ordered and re-displayed. Hirata further teaches interactive user 
selection for selecting a subset of images to be re-ordered and re- 
displayed based on the mutual similarity. 
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The foregoing assertions by the Examiner are respectfully traversed, as the 
Hirata patent, considered alone, or in combination with the various secondary 
references relied upon by the Examiner, fails to teach or suggest all features set 
forth in Appellants' independent claims 1 and 17. The image matching based on 
boundary, as taught by the Hirata patent, does not redisplay a portion of the images 
as claimed, and does not suggest redisplay of a portion of the images first displayed. 
Rather, the first and second similarity calculations disclosed by the Hirata patent are 
a mere sequence of calculations in a method for querying a database. 

Moreover, a similarity matching among candidate images resulting from the 
Hirata patent's region-based and boundary-based matching does not produce a 
redisplay of a portion of the already displayed images for retrieving. The Hirata 
patent does not disclose a first visual display of a plurality of images, and 
redisplaying a portion of the first displayed images, the portion being in reference to 
the first display. 

The Examiner asserts that the displaying and re-displaying of resulting 
candidate images are interactive because users can select the number of the 
candidate images. However, the selection criteria are not based upon visual 
inspection of the first display and defining a portion of the first display for redisplay. 
The Hirata patent's clustering does not cure the noted deficiencies, because the 
clustering is merely a process in the second similarity calculation, but does not teach 
or suggest at least the claimed feature of redisplaying a portion of the first displayed 
images. The Hirata patent does not teach or suggest at least these salient claim 
features. 
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For at least the reasons discussed above, it thus respectfully submitted that 
the claims are in condition for allowance. 



VIII. Conclusion 

For the reasons discussed above, appellants respectfully submit that the 
Examiner's decision finally rejecting Claims 1-20 should be reversed. 

Respectfully submitted, 

Burns, Doane, Swecker & Mathis, l.l.p. 

Date: December 7. 2004 By: 

Patrick C. Keane 
Registration No. 32,858 



P.O. Box 1404 

Alexandria, Virginia 22313-1404 
(703) 836-6620 
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CLAIMS APPENDIX 

The Appealed Claims 

Claim 1 . A method of visualizing and retrieving a data file from a set of 
data files comprising: 

displaying a plurality of images representing corresponding data files on a 
display device using a first distance metric between each data file; 

redisplaying a portion of the images on the display device using a refined 
distance metric; and 

performing at least one of retrieving, marking, and selecting at least one 
desired data file. 

Claim 2. The method of claim 1 , further comprising: 

repeating the redisplaying step until a desired data file is identifiable. 

Claim 3. The method of claim 1 , wherein the first distance metric is 
calculated by a method comprising: 

computing a feature vector for each data file; and 

calculating the first distance metric between each data file using a first subset 
of data contained in the feature vector. 
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Claim 4. The method of claim 3, wherein the refined distance metric is 
calculated by a method comprising: 

calculating a second distance metric between each data file using a second 
subset of data contained in the feature vector which is greater than the first subset. 



Claim 5. The method of claim 4, wherein the step of computing the 
feature vector comprises: 

computing the feature vector for each data file before starting the method; 

storing the feature vector for each data file; and 

accessing the feature vector for each data file. 

Claim 6. The method of claim 4, wherein each feature vector has a length 
of at least eight. 



Claim 7. The method of claim 4, wherein the feature vector includes at 
least one of a color feature and a texture feature. 
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Claim 8. The method of claim 4, wherein the feature vector includes at 
least one of a color histogram, color moment, color coherence histogram, 
Multiresolution Simultaneous Autoregressive (MRSAR) Model, coarseness, and 
directionality. 

Claim 9. The method of claim 1 , wherein the first distance metrics are 
mapped into an N-dimensional space using FastMap for displaying and wherein the 
refined distance metrics are mapped into an N-dimensional space using FastMap for 
redisplaying. 

Claim 10. The method of claim 9, wherein N is two or three. 

Claim 1 1 . The method of claim 1 , wherein the data files are image files. 

Claim 12. The method of claim 1 , wherein the data files are video files. 

Claim 13. The method of claim 1 , further comprising: 

establishing a fixed scale that spans a maximum distance between the 
plurality of data files; and 
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indicating a relative position on the fixed scale for the redisplay of the portion 
of the images, thereby providing the user with a reference frame. 

Claim 14. The method of claim 1 , wherein the fixed scale is at least one of 
a linear scale, a logarithmic scale, and a hyperbolic scale. 

Claim 15. The method of claim 1 , further comprising: 

providing a display depth indication that represents an amount of overlapping 
of images on the display; and 

scrolling to view images that were previously not viewable due to overlapping 
of the images. 

Claim 16. The method of claim 1, wherein the portion of the images 
redisplayed is graphically selected by the user. , 

Claim 17. A method of interactively retrieving a data file from a set of data 
files in real time comprising: 

displaying a plurality of images, each image corresponding to a data file, on a 
display device using a first distance metric between each data file; 
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interactively selecting, by a user, a portion of the images; 



redisplaying the portion of the images in real time on the display device using 
a refined distance metric; and 



retrieving a desired data file. 



Claim 18. The method of claim 17, wherein the first distance metric is 
calculated by a method comprising: 

computing a feature vector for each data file; and 

calculating the first distance metric between each data file using a first subset 
of data contained in the feature vector. 



Claim 19. The method of claim 18, wherein the refined distance metric is 
calculated by a method comprising: 

calculating a second distance metric between each data file using a second 
subset of data contained in the feature vector which is greater than the first subset. 

Claim 20. The method of claim 19, wherein the first distance metrics are 
mapped into a N-dimensional space using FastMap for displaying and second 
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distance metrics are mapped into a N-dimensional space using FastMap for 



redisplaying. 
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EVIDENCE APPENDIX 
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RELATED PROEEDINGS APPENDIX 



